The Kafka-connect container is configured to monitor topics using regex: "topics.regex": "nrt_[a-zA-Z_]+". The issue with specifying a regex is that only topics that exist at the time of connector initialization will be monitored. This causes a problem because certain topics do not exist until the post-processing-service starts. A temporary workaround is in place where the sink connector is restarted during the test execution.

@mpeels can we put a depends_on clause for liquibase to have completed before kafka-connect can start up?

The Kafka-connect container is configured to monitor topics using regex: "topics.regex": "nrt_[a-zA-Z_]+". The issue with specifying a regex is that only topics that exist at the time of connector initialization will be monitored. This causes a problem because certain topics do not exist until the post-processing-service starts. A temporary workaround is in place where the sink connector is restarted during the test execution.

@mpeels can we put a depends_on clause for liquibase to have completed before kafka-connect can start up?

@ericbuckley We can but that won't solve this issue. In this specific case the person-service creates the nrt_patient topic when it processes a message from the nbs_Person topic. So the topic doesn't exist until our test inserts a patient and its processed by the person-service. The Kafka sink connector never detects the new nrt_patient topic and does not start consuming messages unless it is restarted.

I believe (not tested) that if we change from the regex nrt_ to a list of specific topics to monitor, it will fix the issue. Otherwise we will need to continue to have some mechanism for restarting the connector.

@mpeels it looks like the default value for metadata.max.age.ms is 5 minutes. I believe that means it only scans for new topics every 5 mins. I wonder if we lower that in the kafka-connect container, if it will look for new topics more often?

environment:
  CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY: "All"
  CONNECT_CONSUMER_METADATA_MAX_AGE_MS: 1000